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ABSTRACT 



SAFEchild is a non-profit community service agency providing 
parenting information, education, and support to parents, children, and the 
community in the prevention of child abuse and neglect. This evaluation 
provides information to SAFEchild on the impact of the Nurturing Program on 
past participants. Staff were interested in determining participants' 
satisfaction with the program, their retention of learning over time, and 
their ability to apply what was learned to difficult parenting situations. A 
survey was developed and administered using standardized telephone 
interviewing of 26 individuals; 18 were previous participants and 8 were on a 
waiting list to enter the program. The items on the survey involved four 
problem-solving vignettes. Major findings were generally positive and 
included: (1) participants were consistently able to suggest more positive 

parenting strategies for difficult parenting situations than those on the 
waiting list; (2) participants reported an increase in self-esteem since 
beginning the program; (3) both groups were equally able to identify 
children's physical and emotional needs, developmentally appropriate 
discipline strategies, and emotions; (4) all participants expressed 
satisfaction with the Nurturing Program and wpuld recommend the program to 
friends and relatives; and (5) content areas within the curriculum could be 
modified in subsequent sessions to focus on negotiating with children, 
identifying and changing unhealthy and unproductive ways of handling stress 
and resolving conflict, and identifying physical forms of discipline as 
abusive. The researchers caution that the small sample size may have 
influenced the outcome of the survey. (EH) 
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SUMMARY 



An Evaluation of the Nurturing Program at SAFE c/ii/d 



PURPOSE OF THE PROGRAM EVALUATION 

The purpose of this evaluation is to provide information to SAFEo^ on the impact of the 
Nurturing Program on past participants. Staff were interested in determining participants’ 
satisfaction with the program, participants’ retention of learning over time, and participants’ 
ability to apply what was learned to difficult parenting situations. The evaluatiop project 
was undertaken as a community service project by graduate students in the Department of 
Political Science and Public Administration at North Carolina State University. 

EVALUATION DESIGN 

A survey instrument was developed, aligned to the goals and objectives in the program, 
and validated for content (through expert review). The items on the instrument included 
four problem-solving vignettes. Using standardized telephone interviewing, twenty-six 
individuals were interviewed. Eighteen (28%) of those individuals were program 
participants and eight (64%) were individuals on a waiting list to enter the program. A large 
number of past program participants could not be reached by telephone due to outdated 
telephone numbers. Due to the small sample size obtained, the results should be viewed 
with optimistic caution. 



MAJOR FINDINGS 

The findings were generally positive and the evaluation findings should be useful in 
modifying particular areas of the curriculum. 

* Program participants were consistently able to suggest more positive parenting 
strategies when given difficult parenting situations (i.e., vignettes) than those on the 
waiting list. 

* Participants reported an increase in self-esteem since beginning the program. 

* Both groups (participants and those on the waiting J^^were equally able to identify 
children’s physical and emotional needs, developm^fitglly appropriate discipline 
strategies, and emotions. 

* All participants (100%) expressed satisfaction with the Nurturing Program and all 
(100%) would recommend the program to friends and relatives. 

* Content areas within the curriculum to focus on in subsequent sessions include 
negotiating with children, identifying and changing unhealthy and unproductive ways of 
handling stress and resolving conflict, and identifying physical forms of discipline as 
abusive. 

The findings of the evaluation should be of use to SAFEofe^ given the eagerness, 
enthusiasm, aj^ general receptivity towards the evaluation process. The team would like 
to thank «flrie Menestres, Executive Director, and Bettie Murchison, Program 
Coordinator^ffiheir interest in the evaluation and commitment to improving the Nurturing 
Program at SAF ^zlc/uM 
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Context of the Evaluation 



BACKGROUND OF THE NURTURING PROGRAM AT SAFE c/M 

SAFE cAM is a non-profit community service agency providing parenting information, 
education, and support to parents, children, and the community in the prevention of child abuse 
and neglect. One of the many programs offered by SAFEo^ is the Nurturing Program. The 
Nurturing Program is currently a 15-week educational program targeted for parents with children 
from four to twelve years of age. As participants in the Nurturing Program, a group of parents 
and their children attend weekly evening sessions for four months. The primary purpose of the 
program is to teach parents who are at-risk for abusive parenting behaviors to use positive 
discipline strategies. Parents, partners, and children are all encouraged to attend the program; 
facilitators work with different age groups of children in separate play areas. The children leam 
many of the same concepts presented to parents through the medium of play, art, role play, 
puppets, and music. Parents are asked to practice skills taught in the program and to complete 
homework assignments and to report their success in subsequent meetings. 



PURPOSE OF THE EVALUATION OF THE NURTURING PROGRAM 

The purpose of this evaluation is to provide information to staff at SAFE<>&^ on the impact of 
the Nurturing Program on past participants. For example, staff are curious as to what 
participants have learned in the program, what knowledge they retain over time, and which 
techniques they are able to apply in actually disciplining their children. To provide this 
information, a follow-up study of past participants was designed. The evaluation was 
conducted by graduate students enrolled in PA-616 (Program Evaluation) in the Department of 
Political Science and Public Administration at North Carolina State University. The team of 
students involved with the project included: Wendy Combs, Janice Reeth, Pam VanDyk, Olga 
Herrera, and Ron Hasinger, who are all enrolled in graduate programs in Psychology, Public 
Administration, or Educational Research. The experience designing and implementing this 
evaluation served as a valuable practicum experience for the team. 
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Evaluation Design 



A telephone survey of past program participants was used to assess participants’ learning 
and application of knowledge acquired through the Nurturing Program. A group of 
individuals on a current waiting list for the program was used as a comparison group. 

The instrument, which was developed specifically for the purpose of evaluating the 
Nurturing Program, was revised several times before the final version. The instrument was 
developed after a review of the literature (See Appendix A) and critique of the instruments 
currently in use in the Nurturing Program. Before conducting the interviews, the team 
participated in approximately eight hours of telephone survey training, including readings 
and practice administering the survey. 

Following the training, four team members conducted the interviews individually. Since a 
letter explaining the study had been mailed to potential respondents from SAFE cAM, the 
interviewers were well received by those respondents who were reached. Due to wrong 
telephone numbers, difficulty reaching individuals, and a few refusals to participate in the 
evaluation, the final sample size included only 26 respondents (from 76 total). The table 
below identifies the percentages of potential respondents who could not be contacted for 
various reasons. 



Number of 

Outcome Code Potential Percentage 

Respondents 


More than 3 calls 
with No Answer 


5 


7% of total 
sample (N=76) 


More than 3 calls to 
Answering Machine 


6 


8% of total 
sample (N=76) 


Disconnection 


14 


18% of total 
sample (N=76) 


Wrong Number 


13 


1 7% of total 
sample (N=76) 


Refusal 


3 


4% of total 
sample (N=76) 
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On average, it took two phone calls to reach those individuals who ended up participating 
in the study. It is estimated that another 13% of the sample could have been contacted to 
participate in the study if more time for interviewing had been available (i.e., more time than 
the two week calling window used in the evaluation). 

In sum, 28% of past program participants (n = 18) and 66% of those on the waiting list (n = 
8) were interviewed. The survey response forms (See Appendix D) and call records (See 
Appendix E) were then compiled. A codebook (See Appendix F) and Excel data sheet 
(See Appendix G) were used to code respondents’ answers. The responses of past 
program participants and those on the waiting list were then compared. 



For more information as to how the curriculum of the Nurturing Program was reviewed, refer to 
Appendix B. The appendix also provides a rationale for the use of Kirkpatrick’s training 
evaluation taxonomy in the conceptualization of the survey instrument. 



Report Page 6 



Section 

3 



Findings 



For ease of understanding, the results of the telephone interviews are presented in the order in 
which they are included on the survey instrument (See Appendix C). Respondents who had 
participated in the Nurturing Program over the past two years were considered ‘participants’ 
while those on the waiting list were considered ‘non-participants.’ 

PROBLEM SOLVING VIGNETTES 



Four vignettes were included in the survey to assess 
respondents’ knowledge of positive discipline strategies 
and ability to apply these strategies to actual parenting 
situations. The first vignette presents a situation in 
which a child refuses to go to bed; the child runs 
around the house pleading with the parent (See side 
bar). In an open-ended question, respondents were 
asked what they would do in this situation. 
Respondents’ answers to this vignette are illustrated in 
Figure 1 below . 



“At Bedtime, Tanessa’s eight 
year old son refused to go to 
bed. Each night they go 
through the same routine. 
Tanessa tells him to go put on 
his pajamas and get ready for 
bed. Instead, her son runs 
around the house and pleads 
not to go to bed. Tanessa 
gets annoyed with this every 
night. What would you do in 
this situation?” 



Figure 1. Type of Discipline Strategy Suggested in Response to Vignette #1 [Question 4a.] 
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In Figure 1, the majority of participants (78%, n=14) suggested using a positive discipline 
strategy in this situation with the child who refused to go to bed. Strategies which the 
respondents gave which were in accordance with the concepts presented in the Nurturing 
Program at SAFEc^ were considered to be “positive discipline strategies.” In contrast, half of 
the non-participants (50%, n=4) suggested using a positive discipline strategy in this situation. In 
terms of the specific strategies suggested by parents, the majority of participants (n=6) 
suggested giving the child advance notice or a warning that bedtime was approaching (See 
Table 1 below). Many of the participants (n=5) also suggested applying consequences in this 
situation. Note on Table 1 that a second strategy should have been paired with the strategy of 
talking to the child in order for the action to have been considered effective (i.e., merely talking 
to the child with no particular goal or result was considered to be generally ineffective as a 
parenting strategy). 

Table 1 . Number of Respondents Suggesting Particular Strategies to Vignette #1 



4a. Positive Discipline Strategy 


Respondent 


Number 


Advance notice/waming that 


Participants 


n=6 


_^imejs approaching 


_Non : partici pants _ 


n^0_ 


Apply Consequences 


Participants 


n=5 




Non-participants 


n=0 


Use a Reward 


Participants 


n=4 




Non-participants 


n=1 


Time-out 


Participants 


n=3 




Non-participants 


n=1 


Provide a Choice 


Participants 


n=3 




Non-participants 


n=0 


Sharing time 


Participants 


n=2 




Non-participants 


n=2 


Talk with the child (+ one other 


Participants 


n=2 


strategy) 


Non-participants 


n=0 


Diet/Allergy 


Participants 


n=1 




Non-participants 


n=0 


Discussion of Logical 


Participants 


n=1 


Consequences 


Non-participants 


n=0 


Provide a transition period 


Participants 


n=1 




Non-participants 


n=0 



Three non-participants and four participants suggested an ineffective parenting strategy in this 
situation in the first vignette. Ineffective parenting actions were those suggestions which did not 
deal effectively with the issue in the vignette but which were not punitive or inappropriate either. 
One non-participant did suggest an inappropriate behavior in dealing with the situation (“spank 
him”). 

Following the vignette, respondents were asked how the child in the bedtime situation 
might be feeling. The purpose of this second question was to determine the extent to 
which respondents were able to empathize with the child. However, since this vignette 
focused on the child’s behavior (specifically avoiding bedtime), it was not particularly 
useful to ask and hence the results of this item are not presented. 
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In a second vignette, a child, nearly a teenager, 
procrastinates in cleaning her room (See side 
bar). Again, respondents were asked what they 
would do in this situation. Figure 2 illustrates the 
type of discipline strategies which respondents 
suggested in response to this vignette. Roughly 
the same percentage of participants (67%, 
n=12) and non-participants (75%, n=6) 

suggested a positive discipline strategy in 
dealing with the situation (See Figure 2). 



“Maria fights with her twelve year old 
daughter about her room being a mess. 
Maria tells her to clean it but her 
daughter never gets around to it. She 
makes excuses about cleaning the room 
all the time. To Maria, it is so annoying 
that it makes her mad as soon as she 
gets home and sees the room. She ends 
up yelling at her daughter for not doing 
what she told her to. What would you do 
in this situation?” 



Figure 2. Type of Discipline Strategy Suggested in Response to Vignette #2 [Question 5a.] 
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When asked what specific discipline strategies respondents would use in the room cleaning 
vignette, ten participants and five non-participants suggested using consequences with the child 
(See Table 2). Note that these respondents expressed ideas suggesting the use of 
“consequences;” while respondents did not necessarily use the term ‘consequences,’ the 
intention of using consequences was clear. All other specific strategies suggested by 
respondents have been grouped under headings which represent the use of a specific 
discipline technique or idea. The purpose in coding the data this way was to compare the 
number of respondents who expressed the same ideas but who did not necessarily word their 
suggestions the same. 

Five participants and two non-participants suggested an ineffective strategy in this situation, 
such as cleaning the room for the child. One participant suggested an inappropriate behavior in 
dealing with the situation (“I would yell”). 
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Table 2. Number of Respondents Suggesting Particular Strategies to Vignette #2 





Respondent 


Number 


Apply Consequences 


Participants 


n=10 




Non-participants 


n=5 


Use a Reward 


Participants 


n=2 




Non-participants 


n=0 


Talk with the child (+ one 


Participants 


n=2 


other strategy) 


Non-participants 


n=0 


Help or teach the child 


Participants 


n=1 




Non-participants 


n=2 


Discussion of Logical 


Participants 


n=1 


Consequences 


Non-participants 


n=0 


Provide the child with a list 


Participants 


n=1 


of what to do 


Non-participants 


n=0 


Provide a Choice 


Participants 


n=1 




Non-participants 


n=0 


Adjust expectations due to 


Participants 


n=1 


child’s age 


Non-participants 


n=0 



When respondents were asked what they would do in the same situation involving cleaning a 
bedroom with a much younger child, participants and non-participants responded similarly. The 
majority of both groups (89%, n=16 for participants; and 88%, n=7 for non-participants) 
suggested using a modified strategy with the child which was considered to be developmentally 
appropriate for the age of the child. 



In a third vignette, respondents 
were asked what they would do in 
a situation in which a child was 
clearly apprehensive about going 
to school (See side bar). 



“Shanna’s seven year old daughter doesn’t like school. 
Her daughter says the teacher doesn’t like her and 
she can’t do her school work good enough. In the 
mornings, her daughter says she is sick and cries so 
she can stay home from school. What would you do?” 



Participants and non-participants responded similarly to this vignette (i.e., their responses were 
not found to differ). The majority of each group (61%, n=11 participants; and 63%, n=5 non- 
participants) suggested positive discipline strategies in dealing with the child’s apprehension 
(See Figure 3). 
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Figure 3. Type of Discipline Strategy Suggested in Response to Vignette #3 [Question 6a.] 
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The most common strategy suggested by both groups (n=9 participants; and n=5 non- 
participants) was to go to the child’s school and/or talk with the child’s teacher (See Table 3). 
Several participants (n=5) also suggested talking with the child to find out what was wrong. 



Table 3. Number of Respondents Suggesting Particular Strategies to Vignette #3 



6a. Positive Discipline Strategy Respondent 



Number 



Go to the school and/or talk 

wit h_the^hi Id’s teacher 

Talk with the child (+ one other 

strategy^ 

Help the child with school 

woj^homewqrk_ 

Talk to the school counselor 



Check back with the teacher 



Participants n=9 

Non-participants n=5 _ 

Participants n=5 

N<^ -participants n=0 

Participants n=2 

Nqnjjartidpants n=0 

Participants n=1 

N<^ -participants n=0 

Participants n=1 

Non-participants n=0 



When asked to infer the child’s feelings in this vignette (with the child who did not want to go to 
school) at least half of both groups (50%, n=9 participants; and 75%, n-6 non-participants) 
were able to identify a probable emotion. Feelings or emotions considered probable for the 
child in this vignette included: fear, loneliness, feeling unsuccessful, feeling upset, feeling bad, 
not liking school, feeling overwhelmed, feeling insecure, having low self-esteem, feeling unliked, 
feeling that no one cares, feeling intimidated, feeling misunderstood, frightened, worried, 
resentful, and/or withdrawn. Some of these concepts were reworded and hence are not 
presented verbatim. In contrast, improbable emotions for the child, which some respondents 
identified, were “manipulation,” “attention seeking,” “fatigued,” and "controlling.” The participants 
mentioned these improbable emotions. Surprisingly, six participants and two non-participants 
were unable to identify the child’s feelings or emotion. 
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The fourth and final vignette posed to 
respondents was a situation in which a child 
tended to have temper tantrums when asked 
to do something he did not want to do. In 
response to this vignette, two thirds of the 
participants (67%, n=12) suggested using a 
positive discipline strategy (See Figure 4). In 
contrast, half of the non-participants (50%, 
n=4) suggested an ineffective strategy in 
response to this vignette. 



“Casey’s son is six years old. When Casey 
tells his son to do something, his son 
ignores him. When Casey repeats the 
direction, his son begins to whine and has 
a temper tantrum. Casey doesn’t know 
what to do and he sometimes avoids telling 
his son to do something because he 
doesn’t want to hear the whining. What 
would you do if this were your son?” 



Figure 4. Type of Discipline Strategy Suggested in Response to Vignette #4 [Question 7a.] 
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One participant actually suggested a physically inappropriate action in response to this vignette 
(“I would say I’ll pop your tail”). Additionally, one non-participant responded that he/she did not 
know how to deal with the temper tantrum presented in this situation. The two most common 
positive discipline strategies suggested by participants in this situation were time-out (n=6) and 
applying consequences (n=5) (See Table 4). 

Table 4. Number of Respondents Suggesting Particular Strategies to Vignette #4 



7a. Positive Discipline Strategy 


Respondent 


Number 


Time-out 


Participants 


n=6 




Non-participants 


n=1 


Apply Consequences 


Participants 


n=5 




Non-participants 


n=2 


Provide a Choice 


Participants 


n=3 




Non-participants 


n=0 


Ignore the behavior (+ one other 


Participants 


n=0 


strategy) 


Non-participants 


n=1 


Helping the child (+ one other 


Participants 


n=1 


strategy) 


Non-participants 


n=0 


Talk with the child (+ one other 


Participants 


n=1 


strategy) 


Non-participants 


n=0 


Seek outside help 


Participants 


n=0 




Non-participants 


n=1 
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As is evident in review of Tables 1, 2, 3, and 4, non-participants knew and were able to apply 
fewer parenting strategies (i.e., positive discipline strategies) to the four problem solving 
vignettes than participants. Whereas a majority of participants tended to identify one or two 
specific strategies frequently, each of the non-participants tended to suggest a strategy that no 
other non-participant suggested. 

To examine the number of different parenting strategies respondents had knowledge of, an 
index was created by summing the number of different positive discipline strategies 
respondents suggested across the four vignettes. For example, if a respondent suggested 
using an advance notice of impending bedtime and a consistent sharing time each evening in 
response to vignette #1 and suggested using choices and consequences in response to 
vignette #2, this respondent’s index would be 4 (indicating four different discipline strategies 
used). In contrast, if a non-participant suggested using consequences indiscriminately in 
response to several vignettes, this respondents index would be 1 (indicating one discipline 
strategy in sum). 

The results of comparison of the indexes reveals that two participants suggested seven or more 
different positive discipline strategies in response to the four difficult parenting situations 
presented in the vignettes (See Figure 5). 

Figure 5. Diversity of Positive Discipline Strategies Evident Across Four Vignettes. 




However, the majority of participants (33%, n=6) used 5-6 different strategies across the 
vignettes. The majority of non-participants (50%, n=4) used only 1-2 strategies across the 
vignettes. While the individual findings related to each of the vignettes were not so marked, the 
differences between participants and non-participants have become more apparent when the 
indexes (i.e., summed scores) are compared. This represents one of the most important 
findings in the evaluation of the Nurturing Program. Also of interest, no respondents mentioned 
the use of 'parental disappointment,’ Verbal problem solving’ with the child, or ‘restitution’ as a 
positive discipline strategy; these strategies are presented in the Nurturing Program curriculum. 
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PARTICIPANTS’ SELF ESTEEM 

On a scale from one to five, participants were asked to describe their self-esteem (1) at the time 
of the survey and (2) before they began the Nurturing Program. The mean self-esteem score 
for participants before beginning the program was reported to be 2.61. Participants rated 
themselves somewhat higher in self-esteem at the time of the interview (i.e., after completing 
the Nurturing Program) (mean score of 3.67). In contrast, non-participants reported their self- 
esteem at the time of the interview to be slightly higher than participants had reported (before 
beginning the program); the mean score for non-participants was 3.25. With the small sample 
size, it is inappropriate to make conclusions about these relative mean scores. 

HUMAN NEEDS AND STRATEGIES FOR HANDLING STRESS AND ANGER 



In an open-ended question, respondents were asked to list the needs of infants, children, and 
adults. Both participants and non-participants tended to identify physical and emotional needs 
as opposed to intellectual, creative, or spiritual needs (See Figure 6). 



Figure 6. Respondents’ Identification of Types of Need [Question 10.] 
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All non-participants (100%, n=8) and all but one of the participants (94%, n=17) were able to 
identify specific emotional needs. The emotional needs which were identified included the need 
to be nurtured, loved, supported, accepted, understood, listened to, praised, comforted, and 
respected, to name a few. The physical needs which were identified included: need for food, 
clothing, shelter, cleanliness, sleep, water, and safety. Note that the different categories of 
needs sum to more than 100% because respondents were asked to list at least two needs. 

In a subsequent question, respondents were asked to describe what a parent should do in 
attempting to negotiate with a child. For this item, it was apparent that both participants and 
non-participants did not understand what it means to negotiate with a child. The majority of 
both groups clearly misunderstood the concept of negotiation (61%, n=11 participants; and 
75%, n=6 non-participants) (See Figure 7). 
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Figure 7. Respondents’ Understanding of Concepts of Negotiation [Question 1 1 .] 




Participants Non- 

participants 

Respondents 



The answer many respondents gave was to merely talk with the child and apply consequences 
or give rewards. In order to have expressed understanding of the concept, the respondent 
would have had to have mentioned compromise or listening to the child’s side of the story. Also 
of significance, three participants said that “parents should not negotiate with children.” 

All of the respondents who had been participants in the Nurturing Program (n=18) were 
able to suggest productive ways to manage stress (See Figure 8). 

Figure 8. Respondents’ Suggestions for Managing Stress [Question12.] 
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90 % - 
80 % - 



Q) 70 % - 




Participants 
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□ Unable to Suggest Ways 
to Manage Stress 

ea Suggests 1-3 Productive 
Ways to Manage Stress 

B Suggests 4 or More 
Productive Ways to 
Manage Stress 



H 



Non- 

Participants 



Respondents 



While half of the participants (N = 10) suggested 3 ways to manage stress, the other half 
(N = 10) suggested 4 different ways to manage stress. The results for the non-participants 
were similar, although one respondent was unable to suggest any way to deal with stress. 
Overall, the respondents gave a diversity of ideas for reducing or minimizing stress. Refer 
to Table 5 on the next page for a list of the different strategies suggested. 
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When asked to describe ways to handle anger without physically or emotionally hurting anyone, 
participants were able to suggest more ways to manage anger than non-participants (See 
Figure 9). 



Figure 9. Respondents’ Suggestions for Managing Anger [Question 1 3.] 
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Respondents 



Half of the participants (56%, n=10) were able to list 1-2 productive ways to manage anger and 
the other half (44%, n=8) were able to list three different productive ways to manage anger. In 
contrast, one respondent on the waiting list was unable to suggest a way to manage anger and 
the rest of the respondents (88%, n=7) were only able to suggest 1-2 productive ways to 
manage anger. Refer to Table 5 on the following page for a list of the specific methods 
respondents suggested for dealing with anger. 

Respondents were also asked whether bedwetting, fears, and imaginary friends were a normal 
part of childhood. Only one participant and one non-participant answered ‘false’ to this closed- 
ended question (i.e., answered the question incorrectly). 
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Table 5. Lists of Strategies for Reducing Stress and Handling Anger Productively 



STRATEGIES FOR MANAGING ANGER AND STRESS 


Productive Ways to 
Manage Stress Listed by 
Respondents 


Quiet time for self, walk/drive, call someone to talk to, watch a movie, 
deep breath, slow down, take a bath, read a book, rake leaves, ride 
bike, don’t insist on getting everything done, count, get someone to 
help talk to the child, lay down, find a hobby, time away from 
children, listen to music, cross stitch, keep mind occupied, friends, go 
to my room and relax, talk about it, time for self, do something to 
pamper (self), counseling, exercise, cook, sing, meditate, write about 
it, clean, pray, remove (self) from the situation, sit back/relaxAhink 
about it, stay calm, do things 1 enjoy doing, scream alone, talk to 
spouse, eat well, call a parent, talk about what is causing it, try to 
prevent it 


Unoroductive Wavs to 
Manage Stress Listed by 
Respondents 


Tylenol, ignore it, ignore others, medication, whiskey, eating 


Ideas for Managing Stress 
Presented in the Nurturing 
Program 


Valuing, planning, time management, changing the scene, 
contact skills, creating a relaxing environment, improving 
communication skills, listening skills, assertiveness training, 
fighting fairly, flight, adopting a new attitude, imagination skills, 
regular exercise, diet, listening to your body, and learning 
relaxation skills 




Productive Ways for 
Dealing with Anger Listed 
by Respondents 


Walking away/getting away, stomping feet, count to ten, rake leaves, 
time-out, find something (else) to do, remove self from situation, send 
child to room, give a hug, talk about it later, talk it out, visit friends, 
tell other person you would rather be alone, clean, say to myself 
“ nothing is worth being angry,” exercise, get to the root of the 
situation, calm down then deal with it, talk about solutions to the 
problem, write music, do something to get mind off it, stop and don’t 
say another word 


Unoroductive Ways for 
Dealing with Anger Listed 
by Respondents 


Gritting teeth, sitting in comer, run around the house, bury it until 
later, yelling 


Strategies for Dealing with 
Anger Presented in the 
Nurturing Program 


Angry utter, writing a letter, anger cry and scream in a pillow, 
anger role play, and physical exercise activities 




SATISFACTION WITH THE NURTURING PROGRAM 

Finally, past participants were asked about their general satisfaction with the Nurturing Program. 
All past participants of the program (100%, n=18) who were contacted said that they would 
attend additional sessions of the program, if such sessions were available. All of the past 
program participants (100%, n=18) also said that they would recommend the Nurturing Program 
to friends and relatives. Several participants also gave suggestions for improving the program. 
These suggestions included: 

• Increasing the time allotted to group discussion in the sessions 

18 
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• Presenting more information on adolescent development 

• Preparing for sessions in advance (one participant felt that the facilitator was 
preoccupied with her job and did not have the time to prepare in advance 
sessions) 

• Increasing the number of sessions beyond the 1 5-week program 

• Encouraging more volunteers to work with the children 

• Creating a hotline so that past participants can call and obtain advice 

• Providing a recommended reading list or suggested books for parents 

• Providing sessions on a monthly rather than weekly basis 

• Offering a refresher course 

• Providing transportation to the facility 

• Advertising so that parents are aware of the program before they are 
mandated to attend 

• Increasing the amount of time that parents and children leam together 

• Incorporating a session on coping with sick children 

• Encouraging participants to “bring a friend” to sessions to help more families 
and children 

• Discussing how to apply choices and consequences in children with 
attention deficit disorder 

From these suggestions, it is apparent that past participants seriously considered how to 
improve the Nurturing Program. Further, they provided some excellent suggestions. Two past 
participants also said “every parent should (attend)” the Nurturing Program," because the 
“program was the best (the person had) ever been in. Another person remarked that the 
“facilitator was excellent.” Clearly, participants were unanimously satisfied with the program. 
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Section 

4 



Summary 



SUMMARY OF FINDINGS 



Overall, the results of the evaluation are very promising. A visual summary of the findings 
is presented below in Table 6 



Table 6. Summary of Patterns of Evaluation Findings 



+ 

1) More participants than non-participants 
identified positive discipline strategies in 
response to vignettes #1 and 4 

2) Participants used different positive 
discipline strategies whereas the non- 
participants did not 

3) Participants listed more ways to handle 
anger productively than non-participants 

4) Participants report of self-esteem 
increased 

5) All participants were satisfied with the 
program and would recommend it 

6) Participants provided excellent 
suggestions and ideas on improving the 
program 



1) Both groups suggested positive discipline 
strategies in response to vignettes #2 and 
3 

2) Both groups were able to identify emotional 
and physical needs 

3) Both groups used developmentally 
appropriate discipline strategies 

4) Both groups listed strategies to reduce 
stress 

5) Both groups were able to identify a child’s 

emotion 



1) A few individuals from both groups 
suggested punitive or inappropriate 
discipline strategies, unproductive ways to 
manage stress, and unproductive ways to 
handle anger 

2) Both groups misunderstood the concept of 
negotiation with children 



The groups referred to in the Figure above are the group of past participants and the group 
on the waiting list. As the first attempt at evaluating the Nurturing Program, the evaluation 
yielded valuable information. For example, specific content areas within the curriculum can 
be modified to place a heavier emphasis on negotiation and on the identification of 
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unhealthy and unproductive ways of handling stress and dealing with unresolved anger (in 
order to promote the use of healthy and productive methods), and potentially abusive 
parenting behavior (e.g., hitting, spanking). In addition, past program participants provided 
numerous excellent ideas on how to expand and improve the Nurturing Program. 

CONCLUSIONS 

Several cautions are in order in interpreting the data which resulted from this evaluation. 
First, as mentioned earlier, many past participants of the Nurturing Program could not be 
contacted by telephone; a large percentage of individuals could not be reached because of 
telephone disconnections and outdated telephone numbers. With the difficulty reaching 
potential respondents, the result was a small sample size. The sample size which was 
obtained was not large enough to make conclusions about the effectiveness of the 
Nurturing Program in achieving key objectives. Nonetheless, the evaluation did reveal 
some interesting trends and were generally positive. In light of these findings, the team 
encourages SAFE c/u/d to continue in their evaluative efforts. 

RECOMMENDATIONS FOR FUTURE EVALUATION 

The appendices contained in this evaluation report have been included so that the staff at 
SAFE c/u/d may continue to evaluate the Nurturing Program as well as other programs at 
SAFE c/iiM Appendix H, "How to” Readminister the Survey in the Future,” provides a brief 
explanation of standardized telephone interview training. In addition, the survey instrument 
used in the present evaluation has been modified for future use (See Appendices I and J); 
three potentially problematic items on the original survey were omitted from the 
recommended survey. 

There are several issues to consider before conducting further evaluations, however. 
These issues include minimizing interviewer bias, eliminating systematic error, providing 
standardized telephone interviewing training prior to survey administration, and ensuring 
random assignment of potential respondents to interviewers. To reiterate, it is essential to 
provide interviewers with training and practice in administering the survey. Standardized 
telephone interview methods guard against interviewer bias and systematic error by using 
explicit instructions on how to conduct the interview over the phone and how to probe 
incomplete answers. This issue, and other important concerns, are addressed in Appendix 
H. Any additional information needed on the survey process can easily be found in any 
research methods college textbook. 
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Appendix A 



Literature Review of Available Instruments 

In considering instruments to use in the program evaluation of the Nurturing Program, the 
team initially searched the literature for studies related to child physical abuse and neglect. 
The studies which were found were primarily correlational studies of the life history of 
abusers (e.g., stress and abuse potential; childhood history of abuse and abuse potential) 
(Milner, Robertson, & Rogers, 1990; Milner, 1991; Milner & Robertson, 1990; Caliso & 

Milner, 1992). All of these studies employed the Child Abuse Potential (CAP) Inventory as a 
measure of abuse potential. These studies were empirical studies of potential for abuse 
and subsequently did not address the effectiveness of treatment in mediating the correlates. 

In a second search of the literature for other instruments used in measuring abuse potential, 
a useful article was found. This review article concluded that 83 different attitude 
questionnaires published from 1899 to 1986, including the CAP, had marginal reliability and 
questionable validity for measuring potential for abuse (Holden & Edwards, 1989). Given 
the concern over reliability and validity apparent in the literature, no pre-existing instrument 
was found to satisfactorily measure abuse potential as a tool in the evaluation of the 
Nurturing Program. 

A subsequent review of the training evaluation literature revealed a framework for 
measuring satisfaction with a program, acquired learning, and attitude and behavior change 
(Kirkpatrick, 1996). This model was useful in the design of a specific instrument to measure 
the effectiveness of the Nurturing Program. Since it was not possible to directly measure 
parenting behaviors, the instrument was designed as a self-report survey. Vignettes and 
open-ended questions were used in this instrument to assess participants’ application of 
knowledge and learning; the scenarios in the vignettes represented difficult parenting 
situations and respondents were asked how they would handle the situation. The use of 
these vignettes allowed for more than the measurement of acquired knowledge; rather, the 
vignettes required respondents’ to apply their knowledge and learning. 

An expert in the field of child abuse and neglect, Dr. Haskett (in the Psychology Department 
at North Carolina State University) indicated that the use of vignettes in measuring potential 
parenting behaviors was a promising method for research and evaluation. Dr. Haskett’s 
own research had resulted in the development of “situations” (i.e., vignettes) to measure 
parents’ perceptions of ways to solve parenting problems and problem solving behavior 
(PCPS: Parent Questionnaire Administration). Dr. Haskett’s instrument was used as a 
sample in the development of the instrument for this study. 

During revision, the instrument was subsequently aligned to the goals and objectives of 
SAFE c/uH the Nurturing Program, and the curriculum guides used in the program. Hence, 
the instrument was designed to be an accurate measure of the key elements of the 
program. 
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Appendix B 




Methodology Continued 
Review of Curriculum for Instrument Revision 

The Nurturing Program Curriculum Guide and the Parenting Handbook were read and 
outlined to highlight key learning goals and objectives and training content. 

The goals of SAFE c/u/J, the Nurturing Program, and the curriculum guide used in the 
program are enumerated below: 



SAFE c/u/d goals. 

1 . Empower people to prevent or break the cycle of abuse and neglect in 
their lives, and those of their loved ones, by reducing isolation and 
improving self-esteem. 

2. Improve parenting skills of primary caregivers. 

3. Assist individuals to make informed choices and exercise options for 
themselves and their children 

4. Help children learn how to identify abuse and seek help in a safe place. 

5. Raise community awareness about the importance of nurturing 
childrearing. 

6. Assist other service providers and groups to achieve mutually 
compatible goals around abuse and neglect. 



Nurturing Program goals: 

A. Develop self-esteem in all family members 

B. Build empathic awareness of the needs of others 

C. Learn alternatives to hitting and yelling 

D. Increase awareness of self needs, strengths, and 
weaknesses 

E. Increase family communication and expressiveness 

F. Encourage awareness of the developmental needs of 
others 

G. Promote healthy physical and emotional development 

H. Build family support and cohesion 

I. Learn to have fun as a family 

Curriculum Guide goals: 

To increase participants ' ... 

I. Self-esteem and self-concept 

II. Ability to empathize 

III. Ability to handle anger 

IV. Communication skills 

V. Ability to solve problems 

VI. Ability to make decisions 

VII. Ability to negotiate and compromise 

VIII. Understanding of discipline and punishment 

IX. Knowledge of child development 
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X. Use of behavior management techniques 

XI. Understanding of healthy living (e.g., diet, 
alcohol, drugs) 

XII. Knowledge of stress management techniques 

While the SAFEc/^oals are more abstract, the goals from the curriculum guide are 
more specific. 

An informal interview with the Program Coordinator was also conducted to answer the 
following questions about the program and curriculum: 

1. Has the population of parents served in the program changed over time ? 

Describe any changes. 

2. Has there been a change in the source of referrals over time (i.e., have certain 
agencies referred a larger percentage of parents recently)? If yes, describe. 

3. Have the curriculum and activities used in the program changed over time (e.g., 
was a different curriculum used initially)? If yes, describe. 

4. Is a lesson plan used? How closely does the lesson plan follow the guide? Do 
facilitators “tailor ” the program to groups? 

5. What staff and volunteer positions are involved in the program? Describe staff 
qualifications and volunteer characteristics. How have staff positions changed 

overtime? 

6. Describe a typical evening session of the program. 

7. Is there any other important information we need to know about the program? 

The information served to provide a context for the evaluation. The Program 
Coordinator reported that the program was initially a 23-week program focusing on 
parents with children from birth to five years of age. Over time, the length of the 
program was changed to an 11-week program, to a 12-week program, and to the 
present 15-week program. Also, the age range was changed to a focus on parents 
with children aged four to twelve. These changes in the program were not considered 
to be unusual since the program was modified over time based on community needs. 

After review of the goals in the curriculum guide and discussions with the Executive 
Director and Program Coordinator, it became apparent that four particular learning 
objectives were thought to be most important in the program. These areas included: 

1) increasing participants’ empathy for children 

2) increasing participants’ understanding of developmental stages 

3) increasing participants’ use of positive discipline strategies, and 

4) increasing participants’ awareness of alternative positive discipline 
strategies. 

Given the importance of these key learning objectives, the survey instrument was 
designed to focus on these areas. The items on the survey instrument corresponding 
to each of the goals are indicated below: 

• Increased understanding of positive discipline, including behavior management 
(Survey Items 4a., 5a., 6a., and 7a.) 

• Increased knowledge of child development (Items 14., 5b., and 7b.) 

• Increased ability to handle anger (Item 13.) 
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• Ability to communicate effectively while problem solving (Items 4a., 5a. , 6a., and 
7a.) 

• Ability to negotiate with a child (Item 11) 

• Ability to empathize with a child (Items 4b., 6b., and 10.) 

• Increased knowledge of stress management (Item 12.) 

• Increased self-esteem (Items 9a. and 9b.) 

Training Evaluation Taxonomy 

According to Kirkpatrick’s (1996) taxonomy of training evaluation, an evaluation may 
assess any of five different levels of training impact. These different levels are as 
follows: 

A. Reaction - the assessment of participants’ perception of, or 
satisfaction with, the program 

B. Leaminq -the assessment of knowledge acquired through the 
program 

C. Application of the Learning - the assessment of the application of 
knowledge to the real world. 

D. Impact/Results at the Family Level -the impact of the program on 
families 

E. Ultimate Impact - the extent to which the results achieved at the 
family level impact the system (i.e., community, society) 

The first two levels of impact are the most easy to assess. Not surprisingly, evaluations 
of educational programs typically assess participants’ reactions and learning (i.e., levels 
A and B). The instruments already being used in the Nurturing Program measured 
participants’ satisfaction with the program, participants’ learning in closed-ended 
questions, and participants’ attitudes on a standardized instrument (the Adult- 
Adolescent Parenting Inventory). 

The survey instrument (See Appendix C) designed for the evaluation assessed the third 
level of impact in Kirkpatrick’s model (i.e., the application of what was learned to real 
world problems). Hence, the vignettes on the instrument measured more than mere 
knowledge. 

Another reason for using the vignettes concerned the face validity of the items. Face 
validity refers to whether the instrument measures what it appears to measure. For 
example, it is readily apparent to respondents that knowledge-based items measured 
knowledge (i.e., these items are familiar to respondents as test items). In contrast, the 
purpose of the vignettes is not readily apparent to the respondent and hence the 
respondent is more likely to answer with a genuine response. Further, without knowing 
what the vignettes were attempting to measure, the respondent would not be as aware 
of the socially desirable response to give (i.e., the socially appropriate response). In 
sum, the vignettes required respondents to think about the situation presented in the 
vignette and to apply the knowledge acquired in the Nurturing Program to the scenarios 
presented in the vignettes. 
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Appendix C 



Original Telephone Survey Instrument 

Refer to the next three pages for the survey instrument used in the evaluation. 
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I have just a few questions about your family. 





</) 

0 

03 

TO 

u 

0 

£ 

0 

ro 

ra 



TO 



=#: 

3 

5. 

& 



.Q 

m 



c^- 

0 

E 

o 

jz 



o 

>1 

c 

SI 



£ 

■Q 



JZ 

o 



03 

E 

1 

X 



<0 

m 



c^- 

0 

E 

o 

jz 



o 

>> 

c 

O) 

£ 

> 



■O 

03 



CD 

jz 



03 

£ 

CD 



CO 



E 

03 

v_ 

O) 

o 



03 

C 

3 

r 

3 



03 



TO 

I 

\n 

E 

03 

O) 

0 

1 

Cl 

C0 

c 

"O 

03 

TO 

g. 

O 

t 

03 

Ql 

03 

> 

03 

3 

O 

>> 

03 

> 

03 



& 

o 

TO 

cu 

s 



oo 

o 



o~ 

03 

•*-* 

03 

9- 

o 

t 

03 

Cl 

=3 

O 

"O 

T3 

CO 

03 

>N 

TO 



TO 

r- 

3 

C/1 

<M 

o 



O 

r- 



CO 

JZ 



co 

03 



E 

CO 

O) 

o 



JZ 

u 



=tt 

s 

t/i 

CD 

>% 



JO 

h- 



c- 

> 

Q_ 

03 

a) 

JZ 



03 

C 

0 

CO 

c 

3 

o 

o 

ra 

c 

o 

1/3 



0) 

Cl 



T3 

03 

03 

a. 

o 

tt 

03 

CL 

v_ 

03 

> 

03 

Z3 

o 

>> 

03 

> 

03 

X 



§ 

Ou 



CO 



CO 

10 



03 

03 

>N 

"co 

JZ 

$ 



to 

oo 

=tfc 

2 

w 

& 

cm 

O 



JQ 

00 



CO 

h- 



g 

a 

M 

<2 

g 

to 

a> 

S3 

cr 

to 

to 

hJ 



co 

00 



Q3 

i— 

X 

X 

o 

i— 

3 

O 

>> 

~a 

rz 

CO 

03 

C 

tt 

03 

Ql 

3 

0 
>> 

CO 

c 

T3 

3 

a 

c 

on 

1 

03 

jQ 

E 

03 

E 

>x 

1 

03 

u_ 

>> 

C 

03 

E 



E 

a> 

O) 

O 

v_ 

Q- 

03 



t 

CO 



T3 ^ 
Q) O'* 

II 

y jz 

+j 

a I 

3 C 

° 9 
^ 00 
C </> 
03 CD 
JZ <* 
03 
JZ 



o 

E 

CO 
\ 

03 

O 

i_ 

CL 

O) 



CO 

c 

o 

CO 

t/3 

03 

in 

TO 

c 

o 

■O 

03 

TJ 

C 

03 

CO 

3 

o 

X 

3 

O 

5 

03 

JZ 

9 

to 

> 

03 

03 

0 

5 



| « 
.9 03 

tS “ 

8. ro 
c 73 

5 ® 
2 -o 
oo c 

O 03 



03 



CU 

<3 

*s 

TO 

CU 

£ 

CO 

M 

00 

o 



o 

CM 



0 

co 

03 

> 

TO 

CD 

1 

T3 

03 

(71 

TD 

C 

03 



E 

TO 

03 

O 

i_ 

CL 

03 



03 

JZ 

"O 

c 

03 

E 

E 

o 

o 

03 

t_ 

Z3 

O 

>> 



Z3 

O 

§ 



> .2 0 

> ti > 


-t— • 

<4— 

O 


s 




s 


TO c 3 




s 


5 


c 


1 ? <?> 0 


3 


0 


0 


0 



& 

o 

*5 

to 

CL 

£ 

to 

M 

co 

o 



CM 



C 03 uo 
03 -J 
03 O ^ 
E ^CM 

— CO 



O 03 



E 

TO 

w_ 

03 

O 



03 

C 

M 

D 

D 



03 

JZ 

4 — ' 

03 

> 

O 

Q_ 

P 



03 

> 

TO 

JZ 



o 

>> 

o 

"O 

CO 

c 

o 

L— 1 
CO 
03 
03 
03 



TO 

JZ 



>. XI 

- E 

£ 03 


C 

O 


_> 


U 

03 


T3 


0 

>. 


Q 


-*— < 


Zt c 


03 


0 
4 — < 


c 


C 


‘l 


TO 

JZ 


\_ 

3 


$ 




0 


0 


O 


c 


0 


03 


0 

on 






JZ 

0 


■O 






3 


3 


3 


O 


O 




5 


5 


03 


3 


— 




0 



D 03 ^ 

C QQ ^ 
TO M 

Z ro E 

O V) -J 

03 03 ^ 

u- ^ 

^ To 03 
CO 03 c 
C “ 



— 03-1= 
TO Qu 

£ 03 TO 
TO c JZ 
O 



- 03 
TO > 
5 TO 



*o 

H— 

c 

o 

o __ 

03 O — 

1_ -M 

TO -O -q 
lO 03 2 
tZ JZ 
O O 
CL 03 



03 

5 _ 

</) Q3 
C vZ TO 

TO 03 CO 

m n 
D TO 

O • — <<i \ 

>> 5 .2 



03 

JZ 



t/3 03 

^ CD 



J T3 O 



O 

>, 



i a. 



?^.o 
: O m 



O- 

o 

s 

CO 

CL 

£ 

CTJ 

M 

00 

o 



CM 

CM 



5 • ^ 

on ^ 

TO m 9 

TO £ TO 
^ O C 

>> o 

-Q Q. o 

5 c 03 
03 O </3 
‘5 TO 

l. TO 03 
d) 3 cl 
c ro . 

— > </i 

i/i d) ^ 

£ 55 

T3 £ 2 

0 £ ® 
o S £ 



03 



— "D 



— m TO 



X 

3 

O 



TO 

V, co 

o. 2 

O 03 

° C 
U •- 



CO 

CO 



I O- 
TO 

CO 



03 

s 

, I 

% 

£Z 

< 



a 

1 

s 

oc 

co 



13 

, o 

I 

TO 



o 

p 



TO 
I JC 



a; 

E 



I T3 

5 

o 

|X 



on 
, TO 
JD 
O 



O) 

CO 




Appendix D 



Original Response Form 

Refer to the next three pages for the response form used in the evaluation. 
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Appendix E 



Call Record 



Contact 

Attempt 


Date 


Interview 
Start Time 


Interview 
End Time 


Outcome 

Code 


Interviewer 


1 












2 












3 












4 












5 












6 












NOTES 



Outcome Code: 



NA = No Answer 
CB = Call Back 
Cl = Completed Interview 
RF = Refusal 



BZ = Busy Signal 
AM = Answering Machine 
DS = Disconnected 
WN = Wrong Number 
NH = Not Home 





Appendix F 



Codebook for Telephone Survey 

Item Number 

1 . Respondent ID Number 

A 2-digit number between 1-76 which was assigned - e.g., 1 would be 
coded 01 . 

2. Participation in the Nurturing Program [SA2 Variable code "PARTIC"] 

1 = Participant 

2 = Non-Participant 

3. Curriculum 

Not coded due to too few respondents in 23 week curriculum category. 

4a. Vignettes:- Type of Strategy [SA2 Variable code "V12TRAT"] 

1 = Positive discipline strategy ( in accordance with the positive 

discipline strategies presented in the Nurturing Program curriculum) 

2 = Discusses the situation but does not suggest any strategy 

3 = Ineffective but not punitive disciplinary action 

4 = Responds “I don’t know”] 

5 = Punitive or verbally inappropriate response 

4b. Empathy [2A2 Variable code "VIEMPATH"] 

Item deleted because the vignette focused on the child’s behavior as 
opposed to emotion. 

4c. Specific Positive Discipline Strategy: 

1 st column = Reward [2A2 Variable code "V1REWARD"] 

2nd column = Time-out [2A2 Variable code "VtTIMEOU'l 

3rd column = Diet/allergy [2A2 Variable code "VtDIET'l 

4th column = Advance warning about bedtime [SAS Variable code "VtWARN"] 

5 th column = Sharing time [2A2 Variable code"V12HARlN'1 

6th column = Choice [2A2 Variable code "VICMOICE'I 

7th column = Talking [2A2 Variable code "VtTALK'l 

8th column = Consequences [SAS Variable code "VtCONSEQ"] 

9th column = Logical consequences [2A2 Variable code "VtLOGCON'l 
10th column = Transition period [SAS Variable code "V1TRANS"] 

5a. Vignettes- Type of Strategy [SA2 Variable code "V2STRAT"] 

1 = Positive discipline strategy 

2 = Discusses the situation but does not suggest any strategy 

3 = Ineffective but not punitive disciplinary action 

4 = Responds “I don’t know” 

5 = Punitive or verbally inappropriate response 
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Younger Child - Type of Strategy [2AS Variable code ''V2YOUMG'! 

1 = Developmental^ appropriate strategy 

2 = Developmental^ inappropriate strategy 

3 = Responds “I don’t know” 

5 = Punitive or verbally inappropriate response 

Specific Positive Discipline Strategy 

1 st column = Talking [2A2 Variable code ''V2TALK"] 

2 nd column = Consequences (2A2 Variable code "V2CON2E'1 

3 rd column = Logical Consequences (2A2 Variable code "V2LOGCON'1 

4th column = List to do (2A2 Variable code"V2U2T"] 

5th column = Reward [2A2 Variable code”V2REWARD'1 
6th column = Choice [2A2 Variable code "V2CHOICE"] 

7th column = Help/teach the child [2A2 Variable code ''V2HELP"] 

8th column = Adjust expectations [2A2 Variable code "V2EXPECT"] 

Younger Child - Specific Strategy 

1 st column = Talking [2A2 Variable code ”V2dTALK"] 

2 nd column = Consequences [2A2 Variable code ”V2dCON2E'1 
3 rd column = List to do [2A2 Variable code "V2dU2T'l 
4th column = Reward [2A2 Variable code”V2dREWAR"] 

5th column = Choice [2A2 Variable code ''V2dCHOlCj 

6th column = Help/teach the child [2A2 Variable code "V2dHELP"] 

7th column = Time out [2A2 Variable code ”V2dTIMEO'l 

8th column = Adjust expectations [2A2 Variable code ”V2dEXPECT'1 

Vignettes - Type of Strategy [2A2 Verieble code ”V22TRAT'1 

1 = Positive discipline strategy 

2 = Discusses the situation but does not suggest any strategy 

3 = Ineffective but not punitive disciplinary action 

4 = Responds “I don’t know” 

5 = Punitive or verbally inappropriate response 

Empathy [2A2 Vorfoble code "VSEMPATH'I 

1 = Identifies a probable emotion (Concepts: afraid, lonely, fear of 

abandonment, feels unsuccessful, very upset, bad, doesn’t like 
school, overwhelmed, insecure, feeling unsure, low self-esteem, 
feels no one likes her, feels no one cares, intimidated, 
misunderstood, frightened, worried, resentful, withdrawn) 

2 = Identifies an improbable emotion (manipulative, attention seeking, 

fatigued, faking, controlling) 

3 = Responds “I don’t know” or cannot express an emotion 
Specific Strategy: 

1 st column = go to school/talk to teacher (2A2 Variable code ”V22CHOOL"] 
2 nd column = talks to counselor [2A2 Variable code''V2COUN'1 
3rd column = talk to child [2A2 Variable code "V2TALK"] 

4th column = checking back with teacher [2A2 Variable code "V2CHECK"] 
5th column = helping with school work (2A2 Variable code"V2HELP") 



7a. 



7b. 



7c. 



8 . 



9a. 



9b. 



Vignettes - Type of Strategy [2A2 Variable code "V42TRAT"] 

1 = Positive discipline strategy 

2 = Discusses the situation but does not suggest any strategy 

3 = Ineffective but not punitive disciplinary action 

4 = Responds “I don’t know” 

5 = Punitive or verbally inappropriate response 

Older Child - Specific Strategy 

Item deleted because of the age inappropriateness of a 10 year with 
temper tantrums. 

Specific Positive Discipline Strategy: 

1 st column = seek outside help [2A2 Variable code''V4SEEK"] 

2 nd column = ignore [2A2 Variable code "V4IGN0RE"] 

3 rd column = help child [2A2 Variable code "V4HELP"] 

4 th column = time out [2A2 Variable code"V4TIMEO"] 

5 th column = talk [2A2 Variable code "V4TALK"] 

6 th column = ask again when calmed down [2A2 Variable code "V4AGAIN"] 

7 th column = choices [2A2 Variable code "V4CH0ICE"] 

8 th column = consequences (e.g., count down) [2A2 V tumble code 
"V4C0N2E1 

Diversity of positive parenting strategies identified in vignettes #4-8 [2A2 
Variable code "DIVER2E”] 

1 = One-two different positive discipline strategies identified in total in 

the vignettes: loss of privileges, being grounded, parental 
disappointment, restitution, time-out in room, problem-solving, 
planned ignoring, explain how to do it — modeling, teaching, 
punishment (always probe to determine if the punishment is 
appropriate), rewards, explaining why he can't do it, provide a 
choice with limited options, etc. 

2 = Three-four different positive discipline strategies identified 

3 = Five-six different positive discipline strategies identified 

4 = Seven different positive discipline strategies identified 

Rating of self-esteem [2A2 Variable code "2ELFE2T1"] 

Note: for improper coding on this item (e.g., when the interview 
accepted a response in between two discrete categories - e.g., 3.5 --, a 
coin was tossed to unsystematically select the response - i.e., the 3 or 
the 4) 

1 = low self-esteem ... 5 = high self-esteem - mean score reported 

Current rating of Self-esteem before participation in the Nurturing 
Program [2A2 Variable code "2ELFE2T2"] 

1 = low self-esteem ... 5 = high self-esteem - mean score reported 




1 st column = Physical (food/eat/fed, clothing, shelter, clean, sleep, taken 
care of, water, safe environment) [2A2 Variable code "PHYNEED"] 

2 nd column = Emotional (nurtured, loved, to be given limitations, warmth, 
support, someone to talk to, acceptance, friends, caring, confidence, 
understanding, self-esteem, laughter, listened to, attention, want 
their way, praise, comfort, fitting in, respect) [2A2 Variable code 
"EM0MEED1 

3 rd column = Intellectual (learning) [2A2 Variable code "INTENEED"] 

4 th column = Spiritual [2A2 Variable code "SPIRNEED'T 

5 th column = Creative (play) [2A2 Variable code "CREAMEED"] 

6 th column = Other/Difficult to code (limits, help, consequences, 

protection, parent/work with/direction) [2A2 Variable code "OTHNEEDO"] 

Steps in Negotiation [2A2 Variable code "NEGOTIAT"] 

1 = “Don’t know”/no answer 

2 = (Understands concept of negotiation) Response suggests that the 

respondent understands that it is important to listen to the child 
and to compromise in negotiation. The following steps of 
negotiation are identified in the Nurturing Program curriculum: 1) 
Determine if there is a difference of opinion, 2) state your views 
and what you think are the views of your child (do not be 
confrontive or critical or blame and do use I statements), 3) ask if 
your impression of the problem and your understanding are 
accurate (listen), 4) offer a compromise and keep negotiating 
until an agreement is reached. 

3 = Talk without understanding of concepts of negotiation (talking to 

child and/or discussing consequences and rewards is not 
necessarily negotiation) 

4 = Respondent comments that a parent should not negotiate with a 

child 

Number of productive ways to manage stress , including: quiet time for 
self, walk/drive, call someone to talk to, watch a movie, deep 
breath, try to relax/sit down, slow down, take a bath, read a 
book, rake leaves, ride bike, don’t insist on getting everything 
done, count, get someone to help talk to the child, lay down, find 
a hobby, relax, time away from children, listen to music, cross 
stitch, keep mind occupied, friends, go to my room and relax, 
talk about it, time for self, do something to pamper (self), 
counseling, exercise, cook, sing, meditate, write about it, clean, 
pray, remove (self) from the situation, sit back/relax/think about 
it, stay calm, doing things I enjoy doing, scream alone, talk to 
spouse, eat well, call a parent, talk about what is causing it, try 
to prevent it. Strategies suggested in the Nurturing Program 
were: valuing, planning, time management, changing the scene, 
contact skills, creating a relaxing environment, improving 
communication skills, listening skills, assertiveness training, 
fighting fairly, flight, adopting a new attitude, imagination skills, 
regular exercise, diet, listening to your body, and learning 
relaxation skills. 





* Unproductive ways to manage stress: Tylenol, keep busy, 
ignore it, ignore others, medication, whiskey, eat. 


13. 


Number of productive wavs to handle anaer. including: walking 

away/getting away, stomping feet, count to ten, rake leaves, get 
away from the problem, calm yourself down/cool off, time-out, 
walk it off, find something (else) to do, remove self from 
situation, send child to room, walk away, give a hug, talk about it 
later, talk it out, visit friends, tell other person you would rather 
be alone, clean, say to myself “nothing is worth being angry,” 
exercise, shop, get to the root of the situation, calm down then 
deal with it, talk about solutions to the problem, write music, do 
something to get mind off it - go out, stop and don’t say another 
word. Strategies suggested in the Nurturing Program were: 
angry utter, writing a letter, Anger cry and scream in a pillow, 
anger role play, and physical exercise activities. 




* Unproductive ways to handle anger: gritting teeth, sitting in 
comer, run around house, bury it until later, yell. 


14. 


Normal aspects of childhood [SA2 Variable code "CHIIDHO") 

1 = True 

2 = False 


15a. 


Number of children [SA2 Variable code "MUM CHI ID"] 

A 1-digit number - e.g., four children is coded as 4. 


15b. 


Various ages of children 

Not coded due to low number of respondents. 


16. 


Other adult in home? [2A2 Variable code ”0 ADULT") 

1 = Yes 

2 = No 




** Note: Questions 17-22 were not relevant for non-participants; hence 
this data should not be coded for non-participants even if it was 
incorrectly collected and reported. 


17a. 


Participated in similar program? 

1=Yes 

2=No 

Coded as number 23 - refer to 23. 


17b. 


Which Program? 

Reported verbatim: MAD class, STEP, Mother Read, Undefined 
parenting program/class 


17c. 


Year of participation in similar program: 

Not coded due to low number of respondents. 


18a. 


Received personal counseling or therapy in past? 1 =Yes, 2 = No 
Coded as number 23 - refer to 23. 




18b. 


Year of participation in personal counseling or therapy: 
Not coded due to low number of respondents. 


19. 


Number of participating family members [2A2 Variable code "FAMATT'1 
A 2-digit number - e.g., three family members in addition to the 
interviewee would be coded 03. 


20. 


Would attend additional sessions? [2A2 Variable code "ADD2ESS"] 
1=Yes, 2=No 


21. 


Would recommend the program? [2A2 Variable code "RECOMM'1 
1=Yes, 2=No 


22. 


Suggestions - Included verbatim: 


23. 


Participated in similar program or personal counseling or therapy? 

[2A2 Variable code "OTHPRO'1 

If yes to either 17a. or 18a., code 23. = 1 for yes. 
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Appendix G 



Excel Data Sheet 

The actual data is not presented in this version of the evaluation report; the sample 
sizes and percentages are however presented for each conclusion in the body of the 
report. 




Appendix H 




“HOWTO” ... READMINISTER THE SURVEY IN THE FUTURE 

A. TRAINING IN STANDARDIZED INTERVIEWING TECHNIQUES 

Although it is not at all easy to carry out a good, standardized survey in which all 
interviewers behave consistently, the procedures for interviewers to follow in 
handling the question-and-answer process in a standardized way are simply stated: 

1 . Read the questions exactly as worded. 

2. If the respondent's answer to the initial question is not a complete and 
adequate answer, probe for clarification and elaboration in a 
nondirective way; that is, in a way that does not influence the content of 
the answers that result. 

3. Answers should be recorded without interviewer discretion; the answers 
recorded should reflect what the respondent says, and they should only 
reflect what the respondent says. 

4. The interviewer communicates a neutral, nonjudgemental stance with 
respect to the substance of answers. The interviewer should not provide 
any personal information that might imply any particular values or 
preferences with respect to topics to be covered in the interview, nor 
should the interviewer provide any feedback to respondents, positive or 
negative, with respect to the specific content of the answers they 
provide. 

B. PROBING OPEN-ENDED QUESTIONS 

The hardest probing tasks for interviewers involve those connected with open- 
ended questions. The interviewer has to make three judgments of any answer that 
is obtained: Does it answer the question? Is the answer clear? Is the answer 
complete? 

Interviewers should use a very small number of probes which are easy to answer. 
First, the interviewer can merely repeat the question. Other than repeating the 
question, the interviewer should use the following nondirective probes: 

• How do you mean that? 

• Tell me more about that. 

• Anything else? 

The interviewer's task is to decide which of these probes is appropriate, which 
involves analyzing the respondent's answer. The four probes, including repeating 
the question, correspond to the four ways in which a respondent's answer may be 
inadequate: 





1 . The response may fail to answer the question or may answer some other 
question. In this case, the interviewer should repeat the question. 

2. The response contains unclear concepts or terms that make its meaning 
ambiguous. In this case, the interviewer should use the probe, "How do 
you mean that?" 

3. The response may not be detailed or specific enough. With this type of 
response, the interviewer should use the probe, "Tell me more about that." 

4. When a perfectly appropriate response has been given but there may be 
additional points that the respondent could make, the interviewer should use 
the probe, "Anything else?" 

PROBING CLOSED-ENDED QUESTIONS 

A closed ended question is one in which the possible reponse alternatives are 
given. For example, the following question is a closed-ended question: “Family 
rules are mainly for a. Parents, b. Grandparents, c. Children, d. All family 
members. When a question asks the respondent to choose an answer from a list 
such as in this item, and the respondent has not done so, the interviewer's job is to 
explain that choosing one answer from the list is the way to answer the question 
(called training the respondent). The interviewer first explains to the respondent that 
the way to answer the question is to choose one answer and then the interviewer 
rereads the question and list of possible answers. 

One common mistake that interviewers make in handling such a situation is to 
accept an answer that does not exactly fit one of the responses. In this incorrect 
method, the interviewer then interprets and codes the answer into a response 
category; in short, the interviewer has picked the answer instead of the respondent. 
Make reference to this error in training so that interviewers know how to handle this 
type of response. 

RECORDING ANSWERS 

The interviewer’s job is to write down the exact answer the respondent gives. The 
key to standardized recording is to avoid interviewer judgment and interviewer 
summaries in order to avoid interviewer effects on what is written down. 

For closed-ended questions, the key interviewer task is to get the respondent to 
choose one answer and then to record the answer chosen. The only possible 
recording error, other than a clerical error, occurs when the interviewer indicates that 
a response was chosen by the respondent when, in fact, it was not. 

The rule for recording open-ended responses to opinion or attitude questions is 
equally clear and simple: interviewers should write down the answer verbatim; that 
is, the interviewer should write down the exact words given by the respondent, 
without summary or omissions. A problem in writing down summaries and 
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paraphrases is that these will inevitable vary from interviewer to interviewer (with 
different interviewers focusing on different aspects of the response). One way to 
minimize such interviewer effects is to reduce interviewer discretion about what to 
record and instead require that interviewers record responses verbatim. While this 
may slow the interview process since the interviewer must record the exact wording, 
it is well worth the extra time. 

LENGTH OF TRAINING 

The length of the training will be related to the extent of content and interview 
techniques covered in the training, as well as the cost of the training. In academic 
and government surveying, basic interviewer training typically lasts from two to five 
days, with telephone interviewers generally trained for a shorter period of time than 
in-person (i.e., face to face) interviewers. Many survey projects however, have used 
less than one day (i.e., two hour lecture) of general interview training. Our team 
spent more than eight hours over time training and practicing to administer the 
survey. One strong suggestion is that interviewers read and discuss these general 
interviewing techniques with a group discussion following. It is also imperative that 
interviewers have the opportunity to ask questions and to practice administering the 
survey multiple times (with feedback). It is easiest to train interviewers in a group so 
that everyone hears the answer to questions raised. Interviewers could also critique 
each other’s practice attempts in order to come to a clear understanding of the 
importance of standardization in the interviewer process. 

RANDOM ASSIGNMENT OF NAMES TO INTERVIEWERS 

An easy and valid approach to random assignment is to systematically sample from 
a list of all possible respondents. First, create a list of parents who completed the 
Nurturing Program and a second list of parents who are currently on the waiting list 
for the Program. Count the number of interviewers (e.g., four interviewers). Next, 
choose a small random number and count that many names from the start of the 
list. After this, simply choose every fourth name on the list (you are selecting the 
fourth name because there are four interviewers; if there were three interviewers, 
you would select every third name after the random start). For example, if the 
random number is 2, then the second parent on the list would be the first person 
randomly selected to participate in the survey. After this person, the sample would 
include the 6th, 10th, 14 th name on the list, and so on. Use the same procedure for 
the second list. This method of assigning names to interviewers should help to 
decrease the effects of interviewer bias and systematic error. 
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Appendix I 



Recommended Telephone Survey Instrument 

Refer to the next three pages for the modified telephone survey instrument; this 
instrument may be used in future evaluations. 
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Recommended Telephone Survey Instrument 
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Appendix J 



Recommended Response Form 

Refer to the next three pages for the modified response form; this form may be used 
with the modified telephone survey instrument in future evaluations. 
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